Grape quality inspection needs the identification of multiple categories of grapes, and there are many scenes such as depth of field changes and multiple strings in the grape images. Grape recognition is ineffective due to the limitations of single pretreatment method. The research objects were 15 kinds of natural scene grape images collected in the greenhouse, and the corresponding image dataset Vitis-15 was established. Aiming at the large intra-class differences and small inter-class of differences grape images, a multi-scale grape image recognition method based on Convolutional Neural Network (CNN) was proposed. Firstly, the data in Vitis-15 dataset were pre-processed by three methods, including the image rotating based data augmentation method, central cropping based multi-scale image method and data fusion method of the above two. Then, transfer learning method and convolution neural network method were adopted to realiize the classification and recognition. The Inception V3 network model pre-trained on ImageNet was selected for transfer learning, and three types of models-AlexNet, ResNet and Inception V3 were selected for convolution neural network. The multi-scale image data fusion classification model MS-EAlexNet was proposed, which was suitable for Vitis-15. Experimental results show that with the same learning rate on the same test dataset, compared with the augmentation and multi-scale image method, the data fusion method improves nearly 1% testing accuracy on MS-EAlexNet model with 99.92% accuracy, meanwhile the proposed method has higher efficiency in classifying small sample datasets.